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The purposes of this study were to extend the 
investigation concerning the limitations of guantitative data, 
explore the need for and use of qualitative data, and to evaluate two 
chemistry courses (CHEM Study and Modern Chemistry, 2nd edition) • In 
his research, the investigator conducted a partial evaluation of 
these two courses and assessed the knowledge about science and 
scientists possessed by grade eleven and grade twelve students in 
Saskatchewan. Thirtyeight teachers were randomly selected to 
participate in the study; 28 agreed to do so. Student data were 
collected via the Nature of Science Scale — form SASK (NOSS) , an 
instrument originally designed to assess the effect of university 
science programs on university students" understanding of the nature 
of science; the Science Process Inventory form D-SASK (SPI) ; and the 
Test on the Social Aspects of Science (TSAS) which deals with the 
interaction among science, technology and society. The use of t tests 
for matched pairs indicated that both chemistry courses had a 
negligible effect on students* knowledge about the scientific 
enterprise. Qualitative data revealed a balance between a gain in 
understanding and an acquisition of misconceptions. Also identified 
were specific ideas which indicated strengths and weaknesses in 
students' understanding about science and scientists. (FEB) 
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ON USING QUALITATIVE DATA TO EVALUATE TWO CHEMISTRY COURSES 



Glen S. Aikenhead 
Associate Professor 
University o£ Saskatchewan 
Saskatoon, Sask., Canada 

INTRODUCTION 

Quantitative data have specific limitations as far as giving 
feedback to teachers and curriculum developers is concerned. A comple- 
mentary or alternative form of evaluation information could enhance the 
quality of the evaluation process. This alternative foim of infomation 
emerges from asking such qualitative questions as: What ideas have 
students learned? What misconceptions did they acquire? What misunder- 
standings have they still retained? 

At last year's NARST annual conference, a paper was presented 
(Aikenhead, 1973a ) concerning: (1) the limitations of quantitative 
data, and (2) an exploration of the need for, and the use of, qualitative 
data. Project Phys ics was partially evaluated in order to illustrate 
the use and limitations of qualitative data. 

The purposes of this study are: (1) to extend the investigation 
reported last year to NARST, and (2) to evaluate two chemistry courses 
(GHEM Study and Modem Chemistry 2nd edition). This evaluation infor- 
mation was used by the Saskatchewan Chemistry Curriculum Committee in its 
formative evaluation of its chemistry program. 
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THE PROBLEM 

The general state o£ understanding science and the scientist 
would seem to be unchanged since the 1950*s (Aikenhead, 1972, pp. 1-2, 
23-26). However, it is not enough to bemoan the fact that the national 
science curricula o£ the 1960*s have had inconsequential effects on an 
adolescent* s realistic understanding of the scientific enterprise.. 
Science educators need: 

(1) a list of specific misconceptions that are generally 
held by a representative sample of students; and 

(2) some ideas as to the impact science classes are now 
having with respect to this type of knowledge. 

Only then can curricula be developed or modified to overcome present 

deficits in students* understanding. 

In Saskatchewan, chemistry students study either CHEM Study 
(Freeman, Prentice -Hall, or Heath editions) or Modem Chemistry (Metcalfe, 
Williams ^5 Castka; 2nd edition) in grades XI and XII. Approximately 65% 
and 51% of the grades XI and XII students enrol in chemistry, respectively. 

This investigation: 

(1) is a partial evaluation^ of CHEM Study and Modem 
Chemistry taught in Saskatchewan in the 1972-73 
school year; and 

(2) assesses the knowledge about science and scientists 
possessed by grade XI and XII Saskatchewan chemistry 
students. 



Because student learning is defined by the three evaluative instruments 
used to collect the data, the evaluation is limited to students' knowledge 
about science and scientists (the content of the three instruments). In 
this sense, the study is a partial evaluation. 
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PROJECT DESIGN 
Sample 

From the Saskatchewan Department of Education's 1971-72 list of 
chemistry teachers (353 total) , 38 teachers were randomly selected to 
participate in the study. Each teacher was contacted by letter and tele- 
phone in September 1972 and 37 agreed to have their chemistry students 
respond to some evaluation instruments. It is assumed that the students 
of these randomly selected teachers represent the population of Saskatchewan 
chemistry students, even though the students themselves were not randomly 
selected. The sample size was reduced to 28 when five teachers did not 
return the pretest responses, three were unable to have their students 
respond to the posttests, and one teacher did not teach chemistry as 
anticipated. (This reduced the number in the urban grade XI CHEM Study 
subsample more than any other group.) The resultant subgroup of 28 
teachers randomly selected from a population of 353 teachers allows one 
to be fairly confident in generalizing one's results to the total population. 
(The similarity between the student sample in this study and the population 
from which the sample was picked is shown in Appendix A.) 

Table 1 shows the distribution of teachers and students among two 
courses, two grades, and two administrative systems (studying the course 
during one semester or over a full year period). Because there were seven- 
teen teachers who taught both grades XI and XII, one may think in terms of 
there being a total of 45 teachers, for the purposes of analyzing student 
responses . 

Table 1 fits about here 
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The number of students was originally 790, but this number was re- 
duced by 7% because of absenteeism for the posttest. Only the students 
who wrote both the pretest and posttest were analyzed. A further 0.8% of 
the students were deleted due to their answer sheets being unusable. 

Instruments 

Three instruments were used to assess the students' knowledge 
about the scientific enterprise. The tests were evenly distributed among 
the students of each teacher. By this method, a maximum of info imat ion 
was obtained in a minimum of testing time (Walberg § Welch, 1967), however, 
the number of students in each subgroup writing one particular test was 
diminished by one-third. 

Students were asked to respond to every test item on a five point 
scale: strongly agree, agree, I do not understand, disagree, strongly 
disagree. The inclusion of the response, "I do not understand," allowed 
a student to indicate that a choice was impossible due to vocabulary or 
syntax problems. This five-choice response format deviates from the 
suggested response fonnat of the original instruments. The reasons for 
making such a change are discussed below. 

A detailed description of the original instruments, including 
their validation and reliability data, may be found elsewhere (Aikenhead, 
1973b). Only a cursory description is presented here. 

The Nature of Science Scale form SASK (NOSS) was developed at 
Stanford to assess the effect of the university's science programs on a 
student's understanding of the nature of science (Kimball, 1965). The in- 
strument has never been validated for a high school population. Therefore 
on the basis of a preliminary study (with second year nonscience university 



students), the difficult vocabulaiy was simplified and one item was dropped 
altogether. The Saskatchewan chemistry students' responses to the NOSS 
form SASK should give valuable data concerning the instrunent's suitability 
for high school students. The NOSS has 28 items. 

"Tie Science Process Inventoiy form D-SASK (SPI) deals with a stu- 
dent's awareness of the activities, assumptions, products and ethics of 
science O^elch, 1966). The form of the SPI usually used by researchers, 
fom D, has a forced-choice response format- -students are forced to agree or 
disagree with each item. In a former investigation (evaluating Project 
Physics) , it was found that the forced-choice response fonnat led to ambi- 
guities for the researcher. One did not know if a student was disagreeing 
with a statement because he really did disagree or whether he disagreed 
because he did not understand what the statement meant (Aikenhead, 1972, 
p. 221). The same ambiguity exists for an agree response. Therefore, a 
tikert type scale (described above) replaced the agree/disagree format of 
form D. Because students are not forced to guess with the Likert type scale, 
the average scores should drop from form D to form D-^SK. (Comparing 
Saskatchewan students with United States Totms was not an objective of this 
study.) The SPI has 13S iteras. 

The Test on the Social Aspects of Science (TSAS) deals with the 
interaction among science, technology and society i t)^e social nature of 
the Scientific enterprise; and the social and political responsibilities 
of scientists (Korth, 1968). Except for one slight alteration, the original 
response format is the same as the one used in this study. "I do not under- 
stand" replaced "uncertain or undecided." The change was made in order to 
force a student to take a stand if he thought he understood the fl^aning of 
the item. The TSAS has 52 items • 



All three tests were marked in the same manner. Each item has a 
•'correct'' response based on the content validity of the instrument. The 
response "agree" or "strongly agree" or the response ^'disagree" or "strongly 
disagree" was considered correct i£ it concurred with the testes key. The 
"I do not understand" response was always marked incorrect, however it was not 
ignored. Each item was later analyzed with respect to the percentage of 
students who agreed with, did not understand, and disagreed with, the item. 

Many o£ the NOSS^s items appear (verbatum or in modified fonn) in 
the SPI or the TSAS. Therefore, one could say that a high degree of content 
overlap exists between the SPI and the NOSS and bet\^reen the TSAS and the NOSS. 

Table 2 shows the tests* quantitative attributes based on the per- 
formance of the students in this study. These data are not intended for a 
comparison of Saskatchewan students with American norms. As mentioned above, 
the change in fonnat for the SPI pre-empts such a comparison, and data for 
the NOSS have not been reported for high school students. It is of passing 
interest, however, that the Saskatchewan means and standard deviations for 
the TSAS are not unlike those of Korth^s survey group (28.30 points and 
4.98; Gallagher, 1969). 

Table 2 fits about here 

The reliability of the three tests is not exactly comparable because 

each test has a different number of items. Ferguson (1966, p. 381) suggests 

a way to transform reliability coefficients (r ) into a theoretical value 

(r^^) based on an 100 item test: 

r = ^*^xx 

1 - l)r^ 

where: k = 100/ (# of items in the test) 
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For example, the theoretical reliability coefficients (based on an extra- 
polated 100 item test) for the SPI, NOSS, and TSAS are: .82, .69, and 
.80 respectively. 

The reliability of the NOSS is much lower than most acceptable 
standards. Together with the fact that there is a high degree of content 
overlap between the. NOSS and the other two tests, this indicated that in 
the future studies one could readily ignore the NOSS and use just the 
SPI and TSAS. 

QUANTITATIVE RESULTS 

Student responses to the three instruments were machine scored 
and then analyzed^ both quantitatively and qualitatively. The quantitative 
analysis is described first. 

The students were grouped according to their year of study (grades 
XI and XII) and the course studied (CHEM Study and ^todem Chemistry) . A 
further dichotomy within each group (semester and full year administrative 
systems) was not feasible due to the limited number of students in the 
sample. However, this comparison was made by regrouping the data into the 
semester/ full year dichotomy and by analyzing the responses a second time. 
The results of this analysis follow the present discussion. Tables 3-6 
show the student achievement based on total test scores. Diagram 1 sum- 
marizes these four tables. 

Tables 3-6 fit about here 



Appropriate computer programs were used for the data analysis at the 
University of Saskatchewan's Computer Centre. 



TABLE 3 



TESTS FOR SIGNIFICANCE BETIVEEN PRETESTS AND POSTESTS FOR 
GRADE XI CHEM STUDY STUDENTS 





Mean 


uam 


t test 

mm 


«^ b 


CnC 


NOBS (N = 39) pretest 
post est 


9.10 
8.87 


-0.23 


0.70 




2.534 
2.678 


SPI CN « 70) pretest 
postest 


93.54 
94.91 


1.37 


1.01 


ns 


11.998 
12.605 


TSAS CN = 89) pretest 
postest 


28.51 
28.88 


0.37 


0.71 


ns 


5.416 
5.641 



^est for significance of the difference between two means for correlated sairples 
probability of the t^ test value occurring by chance alone (cut off value of p is .05) 
Standard deviation 

significant at the .05 level of probability 



TABLE 4 

TESTS FOR SIGNIFICANCE BETIVEEN PRETESTS AND POSTESTS FOR 
GRADE XI MODERN Q-ffiMISTRY STUDENTS 





Mean 


Gain 


t test 


P< 


SD. . . 


NOSS (N " 43) pretest 
postest 


8.81 
8.86 


0.05 


0.11 


ns 


2.234 
2.237 



SPI (N = 37) pretest 89.35 12.962 

4.35 2.69 .02 

postest 93.70 13.069 



TSAS (N = 42) pretest 28.21 4.575 

-0.10 -0.15 ns 

postest 28.12 5.256 
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TABLE 5 

TEST FOR SIGNIFICANCE BETWEEN PRETESTS AND POSTESTS FOR 
GRADE XII CTEM STUDY STUDENTS 







Mean 


Gain 


t test 


P< 


SD 




postest 


y . /o 
9.31 


-0.44 


-1.65 


ns 


2.840 


SPI CN = 


= 91) pretest 
postest 


98.16 
99.67 


1.51 


1.70 


ns 


11.039 
11.388 


TS/j; CN 


= 102)pretest 
postest 


30.61 
31.01 


0.40 


0.88 


ns 


4.806 
5.386 




TABLE 6 

TEST FOR SIGNIFICANCE BETIVEEN PRETESTS AND POSTESTS FOR 
GRADE XII NDDERN OEMISTRY STUDENTS 








Mean 


Gain 


t, test 


P< 


SD 


NOSS (N 


= 25) pretest 
postest 


9.32 
9.52 


0.20 


0.28 


ns 


2.990 
2.773 


SPI (N = 


• 21) pretest 
postest 


98.81 
98.62 


-0.19 


-0.11 


ns 


8.461 
10.874 


TSAS (N 


= 27) p^retest 
postest 


30.74 
30.22 


-0.52 


-0.66 


ns 


5.139 
5.294 
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A vertical bar defines a critical region surrounding an average score. If one 
critical region overlaps with another critical region, then there is no signif- 
icant difference between the two scores (at the .05 probability level). 
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On the basis of total test scores, only the grade XI ^todem Chem- 
istry group showed any significant gains, 4.35 points on the SPI . The 
same group made \ o significant gains on the other t.vro tests. Although 
4.35 points may represent a statistically significant improvement (approxi- 
mately a one-third stmidard deviation), one could still be disenchanted 
with its being a fairly small gain for a test which has 135 items. 

The grades XI and XII CHEM Study group and the grade XII Modern 
Chemistry group made no significant g^in in total test scores. 

Figure 1 fits about here 

When all the data were dichotomized between the semester and full 
year administrative systems (pooling grade XI and XII along with CHEM 
Study and Nfodem Chemistry students) significant changes in total test 
scores resulted for the full year group on the SPI only (pretest 92.29, 
posttest 95.75, gain 3.46, for 101 students). 

QUALITATIVE RESULTS 

Although the data analyzed above yield quantified statements, these 
results are ambiguous. For example, what does a 4.35 point gain on the 
SPI really mean? This ambiguity leads one to ask qualitative questions 
such as: What ideas have jtudents learned? What misconceptions did they 
pick up? What misunderstandiiigs have they still retained? These questions 
are answered by two kinds of item analyses. 

First of all, each item of each test is examined to see if students' 
correct responses significantly gained or dropped over the year. This analysis 

O 

ERIC 



reflects the impact o£ each chemistry course on student learning. Every 
item is analyzed with respect to the changes in student responses between 
the pretest and posttest. There are only two changes possible: (a) from 
an incorrect response to correct response, and (b) from a correct response 
to an incorrect response. The probability is .50 that students who change 
their response move to a correct answer. McNemar (1969) has derived a 
chi square test specifically for this situation: 

I r A with 1 de^ee of freedom''", where A 

/y^ _ — ! ■ ^ " — and D are cell frequencies in the 
( A + D) following contingency table: 



Item X 



Posttest 
1 0 



Pretest 



0 
1 



A 


B 


C 


D 



1 signifies a correct 
response 

0 signifies an incorrect 
response 



The chi square analysis determines which items experience a statistically 
significant change in student response between the pretest and posttest. 

Secondly, the percentage of student agreement, not understanding, 
and disagreement is also analyzed for each item. The results of this 
analysis are discussed below in the sections: "An Analysis of Student 
Correct Responses," and "An Analysis of Student 'I Do Not Understand' 
Responses." 



IVhen A+D<20, a Yate's correction for small frequencies must be intro- 
duced. The equation becomes: >:f'= C|A-D| -1)^ / (A+D) . 
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The Impact of the Chemistry Courses 

Tables 7-14 list the items which experienced significant gains or 
losses. Except for the grade XI Modem Chemistry group, students generally 
gained as much misinformation as they did an understanding o£ the scientific 
enterprise. (This result was also reflected in the lack of significant 
gains or losses in total test scores. Tables 3-6.) 

As shown in Tables 7 and 8, the grade XI CHEM Study students tended 
to learn something about: (1) scientific theories, (2) the interaction of 
science and society, and (3) the interdependence of science and techn,ology. 
They also learned a little concerning limitations to science. Meanwhile 
some misconceptions within the CHEM Study gi'oup worsened. Students in 
both grades XI and XII tended to leam that there is no room for imagination 
or intuition when doing science, and that there are natural phenomena \rfiich 
should not be investigated by scientists. The 11th graders became more 
confused about: (1) the aims of science and the aims of technology, 
(2) the nature of quantitative data, and (3) the use of the simplicity 
value in scientific thinking. 



fables 7 5 8 fit about here 



Oddly enough the grade XII CHB1 Study students increased their 
underscanding of the aims of science and technology (recovering from the 
"mis learning'* of grade XI). As illustrated in Tables 9 and 10, these 
students became more aware of the interaction of science with society 
beyond the tedino logical interface. They also tended to learn something 
about hypotheses and deduction. However, the 12th graders picked up some 
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erroneous ideas concerning a scientist. In addition to the distorted know- 
ledge described in the paragraph above, students tended to increase their 
misunderstanding regarding: (1) the role o£ opinion and fact in making 
scientific conclusions, (2) what a scientific theory is, (3) the assuirption 
of nature's susceptibility to human Ui derstanding, (4) the role of open 
coTTununication , and (5) the practicality value applied to scientific 
knowledge . 



Tables 9 and 10 fit about here 



It was the grade XI Modem Chemistry group which made the significant 
gain in their total test scores. Table 11 shows that these students became 
less naive about: (1) ''The Scientific Method," (2) models duplicating 
reality, and (3) the finality and inalterable truth of scientific knowledge. 
Besides learning more about the relationship between evidence and theory, 
the 11th graders came to see science as less complicated to understand. 
They also realized that scientists themselves do not assume that some 
problems are to complex to be explained. On the other hand, some miscon- 
ceptions were learned. The students tended to learn that (1) scientists 
do not do library research (only high school students?) , (2) serendipity 
plays no important role in science, and (3) theories may not be modified 
in light of new evidence (a paradox to their achievement described above) . 



Tables 11 12 fit about here 
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The 12th grade Modem Chanistry students made the least significant 
gains or losses. As indicated by Tables 13 and 14, these students became 
more aware o£ the causality and consistency assumptions but tended to become 
misinformed concerning induction and deduction • 



Tables 13 § 14 fit about here 

In comparing the achievement of the four groups of students, one 
finds very little similarity in the items which underwent significant gains 
OT losses. Thus, each course at each grade level would soein to represent 
different learning experiences • 

It is also interesting to consider the actual number of items that 
showed a significant change in correct response between the pretest and post- 
test. The present investigation may be compared with a similar study which 
investigated Project Physics and other physics courses (Aikenhead, I973a J • 
Compared with American physics courses, the Saskatchewan chemistry courses 
led to very few significant gains and led to a large number of significant 
losses. 

Any significant CHEN! Study gains tended to relate to the social 
' aspects of science, while any Modem Chemistry gains tended to relate to 
the processes of science. 

An Analysis of Student Correct Responses 

The Saskatchewan chemistry students did have '.i good understanding in 
certain areas of knowledge about the scientific enterprise. These areas 
may be defined by the content of the items for which more than 85% of the 
students responded correctly. (Of course, there may well be items so 



13 

important that one would not be satisfied with anything less than 100% correct 
student response. This decision is left to the reader 0 On the other end of 
the scale, there were definite, widespread misconceptions about science and 
scientists among the students. These may be revealed by the content of 
items for which less than 50% of the students responded correctly. 

The ''above 85-6" and '"below 50%" criteria arc arbitrary. They led to 
the identification of 41 and 65 items, respectively. These two numbers are 
unbalanced due to the NOSS data. No items from the NOSS were above the 85% 
correct response mark, but 23 items were below the 50% correct response 
mark. The SPI and TSAS showed a balance in the number of items above the 85% 
and below the 50% correct response mark. 

A coniplete list of items ineeting the "above 85%" and the "below 50%" 
criteria would be unreasonably long to include in this report. However, 
Appendix B identifies these items and a descriptive siimmary is given below. 
This description of Saskatchewan chemistry students' strengths and weaknesses 
has been organized into several content areas: the aims of science, its 
epistemology p its tactics, its values, its interactions with society, and its 
human needs. This categorization scheme is purely a heuristic one (Aikenhead, 
1972, Chapter 1). The purpose of presenting the data in such a way is to help 
Saskatchewan teachers: (1) to identify students' strengths and weaknesses, 
and (2) to consider possible changes ^^^lich might be made to their 
chemistry classes. 

For the purposes of this general description, all students are analyzed 
as one single group, Saskathcewan chemistry students, because there were 
only minor differences in the students' achievement: (1) between the grade 
XI and grade XII data (with a slight trend toward a higher achievement 
on the part of the 12th graders, and (2) between pretest and posttest data 
(except for those cases already discussed above in connection w\th Tables 
ERJC 7-14). In addition, the only items considered for discussion are 
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those which meet the ''above and 'T^elow 50%" criteria for three 
or four of the original four groups of students (QiBI Study grade XI 
and XII, and Modem Chemistry grades XI and XII). 

Aims of Science 

Nbst students recognized that scientists do formulate theories 
to explain observations. However, the aims of science were hopelessly 
confused with the aims of technology. The students did not understand: 
(1) that the principal aims of science are: (a) to explain natural 
phenomena in terms of principles and theories, and (b) to reduce obser- 
vations to a collection of mathematical relationships; and (2) that 
curiosity and a desire for self-esteem primarily motivate scientists. 

More often than not, students wrongly considered that the primary 
object of basic scientific research is: (1) to provide the people of 
the world with the means for leading better lives, and thus (2) to 
produce new and improved machinery and living conveniences. In summary, 
less than half of the students correctly agreed to the statement, "The 
main object of basic scientific reseach is the discovery of understanding 
rather than its practical applications." 

Epistemology of Science 

There are many facets to the epistemology of science including: 
definitions, assumptions, observations, hypotheses, theories, models, 
laws, and general aspects of scientific knowledge. 

Definitions 

Two -thirds of the students erroneously thought that science was 
simply "an organized body of knowledge.** A majority of students were con- 




fused, or did not understand, the meaning of induction and deduction. 
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Assu mptions 

While most respondents believed scientists assume that "all effects 
in nature have causes," many incorrectly thought scientists assume that "some 
natural thirvgs will never be understood." Students did not know that 
scientists accept assumptions before "they are proven true," and a large 
number disagreed with the consistency assumption. 

Observations 

Students did veiy well in their understanding the nature and role 
of observations in science. Seven items belonged to the "above 85%" 
category while none were poorly done. Carefully recorded, accurate 
observations were seen as inportant features of modem science. Just 
because measurements involve numbers did not mean the measurements cannot 
be im)ng. Students recognized degrees of preciseness in examples of mea- 
surements, and they realized that two people making the "same" observation 
may see different things. 

Hypotheses and Theories 

Most students understood that hypotheses "may be wrong," and that 
theories "may be modified in li^t of new evidence." But when given an 
incorrect example of a hypothesis, these students mistook it to be a 
good exanple. The majority of students also felt that hypotheses have 
more experimental support than theories, thereby revealing an inaccuracy in 
their definition of the terms. 

Models 

A major ambivalence emerged from students' ideas about scientific 
models. Regarding the Bohr model of the atom, over 85%of the students 
recognized it as a '"man-made" convenient representation for the purpose of 
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understanding the atom. On the other hand, these students carried with 
them the misconception that the Bohr model ''pictures the atom as we 
actually know it to exist," because it is "a scaled-up picture of what 
scientists have seen in their microscopes/' Another paradox was discovered. 
Over half of the students wrongly disagreed with the statement, 'Many 
scientific models (like the model of the atom) are made up through man*s 
imagination, they do not duplicate reality;" while over two-thirds 
correctly disagreed with the statement, "Scientific models are exact 
duplications of reality." Sifting through this data, one may conclude that 
students have superficial ideas concerning the nature of scientific models; 
models are indeed man-made but not by man^s imagination. 

Laws 

Again it appears as if a superficial understanding exists about 
scientific laws. Most students correctly believed that when a statement 
becomes a law of science, it can still be changed. However, over half 
of the students affirmed, most incorrectly, that "a physical law is peim- 
anent" and that "the law of conservation of energy is an example of an 
unchanging truth." Perhaps the students believe that statements may change 
but laws do not a very peculiar state of mind. On the positive side, 
most students were not naive enough to believe that nature cannot disobey 
the laws of science. 

General Aspe cts 

On the one hand, many students asserted tiiat scientific knowledge is 
not final but ij in the process of development; but on the other hand, half 
of the students claimed that scientific knowledge is not tentative. The 
"man-made" characteristic of scientific knowledge was grossly misunderstood 
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by most students. Very few agreed that "classification schemes are made up 
by scientists who organize observations of nature to fit these classification 
schemes that is, a classification scheme does not represent an inbom 
quality of the things being classified." Scientific knowledge was also 
misconstrued to be "absolutely true," and most students did not realize that 
facts are the beginning and end points of science. In addition, rock 
collecting was viewed as an examplary scientific investigation. However, 
students realistically understood that scientists do have differences of 
opinion and that theories we believe today were at one time vigorously 
opposed by emminent scientists. 

Tactics of Science 

The notion of "The Scientific Method" is probably one of the most 
widespread and misleading pieces of science education dogma ever conceived. 
Less than 10% of Saskatchewan chemistry students are aware of the fact that 
"scientific method is a myth which is usually read into the story after it 
has been completed." In the minds of adolescents, the discovery of "The 
Scientific Method" has given scientists a definite approved procedure which, 
if followed consistently step by step (usually five or seven) , will solve 
almost any problem or answer almost any question in the realm of nature. 
Not inconsistent with these gross misunderstandings are the correct views 
held by over 85% of the students: (1) that failure can occuir even if one 
rigidly follows "The Scientific Method," and (2) that there are many 
methods of solving scientific problems (presumably within the confines of 
'The Scientific Method") . 

Students fared much better when dealing with scientific experi- 
mentation. For most students, an experiment: (1) "is a set of conditions 
under which observations are made, (2) is used to test hypotheses^ 
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(3) should be repeated, and (4) can validly yield results inconsistent 
with previous answers. At the same time however, a more sophisticated 
description of an experiment was rejected by over half of the respondents, 
"to set up a situation in which the control of variables is greater than it 
is in ordinary course of events/' Painfully, the majority of students 
believed that "experimentation is principally concerned with proving the 
laws of nature." (Does this reflect the manner in which high school labs 
are carried out?) 

Many students understood that: (1) a search of the literature, 
(2) the process of grouping observations, and (5) serendipity; all have a 
role in scientific investigations. On the other hand, a large proportion 
of students discounted intuition or creative imagination as leading to many 
important scientific ideas. Instead, students strongly supported the 
plodding systematic process of logical thought. In addition, they be- 
lieved that large sim^s of money pumped into scientific research is the 
most important requirement for the progress of science. 

Values of Science 

There are many values explicitly or implicitly held by research 
scientists (accuracy, parsimony, open-mindedness , skepticism, honesty, to 
name just a few). Most of the respoiidents agreed that scientists should 
share their findings and discoveries , that this allows for independent con- 
firmation, and that this is essential to scientific progress. However, 
the majority of students incorrectly felt that a scientist's reluctance 
to do so was based on the danger of exposing secret information. Many 
students properly expressed reasonable skepticism of published results but 
thought that "The majority of newly suggested theories are accepted by the 
scientific community." Most students affirmed the value that scientific 
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conclusions should be based on fact, not opinion • 

Some values (objectivity, practicality, simplicity) accepted by the 
working scientist were not understood by the high school students. 'Hiey 
did not realize that objectivity is an unobtainable ideal towards which 
one constantly strives. Instead, the students erroneously believed that 
objectivity was a fact of scientific methodology. Moreover, these adole- 
scents felt that a scientist is likely to be unbiased and objective in areas 
outside of science! A majority of students also held the misconception 
that research scientists believe their discoveries should h^ve practical 
applications (a position consistent with the confusion over tJie aims of 
science and technology). Students do not view discoveries as solving in- 
tellectual puzzles or reducing a scientist's curiosity. 

Few respondents realized that scientists use a sinplicity value in 
their preference for an oqianation or hypothesis. Instead, scientific 
knowledge was thought to be judged highly on a criterion of complexity 
the more complex, the better. (Parsimony is the value underlying the aim 
to formulate a simple set of expressions that explain diverse phenomena •) 

Interaction of Science and Society 

Most students knew that advances in science and technology have 
created, and will create, many changes in our culture, specifically in 
social and economic spheres. A large proportion of respondents believed 
that there must be a channel of communication between the scientific 
community and the lay public in order tliat the latter may make wise 
decisions concerning the former. Keeping the layman informed of scientific 
findings at his level of understanding was seen as crucial. 

However, science was considered: (1) to have little value in helping 
solve today's social problems, and (2) to have no effect on society other 



TABLE 15 

NUMBER OF "I DO NOT UNDERSTAND" RESPONSES TO ITEMS IN THREE TESTS ^ 



Test 



Total number 
of items 



# of itans where 
"I do not under- 
stand" response 
is greater than 



# of itans where 
"I do not under- 
stand" response 
is greater than 



NOSS 

SPI 

TSAS 



28 
135 
52 



2 
7 
1 



1 
20 
6 



Combined sample of all respondents, 
'This includes the numbers in the previous column. 



TABLE 16 

"I DO NOT UNffiRSTAND" RESPONSES TO ITEMS IN THREE TESTS ^ 



Test Items where "I do not understand" Items where "I do not understand' 



NOSS 
SPI 

TSAS 



„ b 



response is greater than 20 



2, 9 

15, 57, 61, 62, 89, 104, 120 



37 



response is greater than 10% 
27 

8, 12, 14, 43, 50, 70, 76, 79 
101, 105, 111, 118, 125 

4, 14, 20, 27, 44 



^Combined sample of all respondents. 

'This does not repeat the items in the previous column. 
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than through the technological implementation o£ scientific knowledge. 
Surprisingly enough, a majority of respondents disagreed that a democratic 
public ''should ultimately control the support of science and the use of 
its achievements." 

Human Needs in Science 

Many students felt, as do many scientists, that "scientists 
should be concerned with the potential harm that might result from their 
discoveries." However, a large proportion of students were unaware that 
"winning the esteem of associates is one of the main incentives of the 
scientist." 

An Analysis of Student "I Do Not Understand" Responses 

Table 15 reflects the degree to which students did not understand 
the items on the three tests. Between 10% and 15% of the test items were 
not understood by at least 10% of the Saskatchewan chemistry students, 
while between 21 and 71 of the test items were not understood by at least 
20% of the students. These items are identified in Table 16. 

CONCLUSIONS 

Quantitative analysis (t^ tests for matched pairs) indicated that 
both chemistry courses had a negligible effect oh students' knowledge about 
the scientific enterprise. Except for the grade XI students studying Modem 
Chemistry C^ho accomplished a statistically significant gain in their awareness 
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of some processes of science), chaiiistry in grades XI and XII had little 
impact on Saskatchewan students' impressions about science and scientists. 
A further analysis of student achievement revealed a slight advantage 
held by a course taught in a full year over the same course taught in a 
single semester. 

The qualitative data supported this result but they also revealed 
far more information. In both chemistry programs, there was a balance 
between a gain in understanding and an acquisition of misconceptions. 
The meagre increase in understanding for the CHEM Study group tended to 
concern some social aspects of science, while the Modem Chemistry group 
slightly increased their understanding of some processes of science. Mis- 
understandings arose concerning, in part: (1) the role of imagination 
and intuition in doing science, (2) the scope of a scientist's investi- 
gat ion into natural phenomena, (3) the aims of science, (4) the nature 
of quantitative data, (5) the use of the simplicity value in scientific 
thinking, (6) the role of serendipity, (7) the modification of theories 
in light of new evidence, and (8) the meaning of induction and deduction. 

Also identified were specific ideas which indicated strengths and 
weaknesses in students' understanding about science and scientists. Some 
of these ideas are described here. Over half of the students erroneously 
thought that: (1) scientific classification schemes are inherent in 
nature, (2) scientific laws are permanent truths, (3) the point of experi- 
ments is to prove the laws of nature, (4) there is such a thing as 
"The Scientific Method," and (5) money automatically buys scientific 
results. Over half of the students did not know that: (1) there are 
limitations to the use of scientific inquiry, (2) scientific models are 
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similes or metaphors and do not duplicate reality, and (3) scientific 
knowledge is tentative. Over half of the students did not understand 
the meaning of induction, deduction, hypothesis, and theory. Over half of 
the students also confused the aims of science with the aims of technology. 
These results were similar to those of recent American surveys (Korth, 1969 
using 1500 Grade XII students in the northwest sector of the United States; 
and Aikenhead, 1973c, using 850 students of a randan sample of American 
and Canadian physics teachers). This recent investigation indicates that 
Saskatchewan students are not unique in their misconceptions about science 
and scientists. 

Apparently the distorted ideas which students hold about the 
scientific enterprise are seldom challenged in chemistry classes. Students 
superficial vieivpoints also escape analysis. 

IMPLICATIONS 

The NOSS form SASK was found to have a low reliability. Several 
items in the NOSS are also found in the SPI or TSAS. For these reasons, 
it is recommended that the NOSS form SASK not be used in investigations 
using high school students. 

The inclusion of the response '^I do not understand'' reduces the 
ambiguity in student responses because there tends to be less guessing. 
Further investigation is necessary, however, to study the effect of dif- 
ferent response foimats on reliability measures and statistical compari- 
sons among group mean scores. A different scoring system (for example, 
2 for a correct response, 1 for a "I do not understand" response, and 0 



for an incorrect response) may also have a noticeable effect on reliability 
and statistical comparisons. 

The Saskatchewan Chemistry Curriculum Committee are far more in- 
formed by the qualitative feedback than they are by the quantitative data. 
There was little apparent change in the average test scores of the CHEM 
Study and >fodem Chemistry groups. However, with the use of qualitative 
data, one can pin point the ideas that were being properly learned as well 
as the misconceptions that were being acquired. Consequently, one can 
understand in what ways (by what knowledge) the two chemistry groups 
differed. The qualitative data give specific feedback which can be 
directly related to: (1) the objectives of the chemistry courses, and 
(2) needed improvements in the courses. 

This study represents a different circumstance from the study 
reported at last year's NARST annual conference (Aikenhead, 1973a). There, 
the Project Physics group showed a statistically significant achievement 
over a non-Project Physics group. Therefore in the previous study, the 
qualitative data clarified the quantitative difference between the two 
groups, whereas in the present investigation, the qualitative data revealed 
differences where the quantitative analysis found none. 

Science educators need a list of specific misconceptions generally 
held by high school students, for only then can curricula be developed or 
modified to overcome present deficits in student understanding. The data 
from this investigation greatly contribute to the validity of such a list. 

Qualitative data appear to enhance the quality of the evaluation 
process. This improvement comes from asking such qualitative questions as: 
IVhat ideas have students learned? What misconceptions did they acquire? 
UTiat misunderstandings have they still retained? 



APPENDIX A 



Grade XI 

OEM Study 
Modem Chemistry 

Grade XII 

aiEM Stud/ 

Modem Chemistry 



Number of Students 
Sample Poplation 

238 8,171 
129 2,717 

301 6,429 
73 1,866 



% of Students 
Sample Population^ 

Rural schools 56% 52% 

Urban schools 16% 35% 

Comprehensive schools 28% 13% 



^figures are for all students in grades X to XII 
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ITEMS FOR WHICH 'fHE AVERAGE STUDENT CORRECT RESPONSES WAS ABOVE 85% AND BELOW 501 



topic 


above 851* 


below 50% 


SPI 


TSAS 


NOSS 


SPI 


TSAS 


Aims of science 


46 




4, 10 


51, 129 


9, 22 

7ft %i 

37 


Epistemology of science: 
definitions 

assumptions 

hypotheses 
theories 

laws 

general aspects 






9, 11 


62, 89 




67 






27, 44 
75, 119 




80 






40 




25 






50 




32, 34 




16 


31, 35 




23, 24 




5 


14, 111 




3, 28 
102, 106 




2, 21 


41, 56 
61, 120 


2 


Tactics of science: 

"The Scientific Method" 

tactics § strategies 


82, 124 
135 




3, 17 
20, 25 
28 


107 


3 


26 , 39 
48, 83 
122, 133 
134 




1, 13 
15 


30, 57 


6 


Values 


5, 6 
55, 60 
102 


15, 36 


6, 7 
8, 27 


9; 64 

S6, 117 


14, 32 
42, 51 


inceracciun oi scxent-e 
and society 




1, 10 
26, 40 
41 






19, 47 
48 


Human needs in science 




5 






11 



No NOSS item met this criterion. 
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